Skip to content

test(northwind): poll for async auth-cache propagation after alter_role#1475

Open
kriszyp wants to merge 3 commits into
mainfrom
kris/flake-northwind
Open

test(northwind): poll for async auth-cache propagation after alter_role#1475
kriszyp wants to merge 3 commits into
mainfrom
kris/flake-northwind

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 24, 2026

Copy link
Copy Markdown
Member

De-flakes northwind.test.mjs, which was failing ~50% of CI runs with varying line numbers due to three stale-auth-cache races.

Root cause: alter_role returns HTTP 200 after the DB write but before async ITC permission-cache invalidation reaches all worker threads. The test assumed the role change would be immediately visible on the next request, but a worker with a stale cache could serve it instead.

Fixes (test-only, no product code changed): Three affected requests are now wrapped in the existing waitFor() poll helper, retrying until the permission change is visibly effective:

  • CSV upsert that requires newly granted attribute permissions
  • Two cross-schema SQL JOINs that require the updated role to be visible

Verified: 5/5 local runs, 560 tests pass.


Generated with Claude Opus 4.8

@kriszyp kriszyp requested a review from heskew June 24, 2026 12:51

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces polling using a waitFor helper in several integration tests to account for asynchronous permission propagation to the worker auth-cache. The feedback suggests ensuring that the returned response from waitFor is explicitly asserted for the expected HTTP status code (e.g., 200 or 403) before inspecting the response body. This prevents silent test failures or unhelpful TypeErrors if the polling times out.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread integrationTests/apiTests/northwind.test.mjs Outdated
Comment thread integrationTests/apiTests/northwind.test.mjs
Comment thread integrationTests/apiTests/northwind.test.mjs
@claude

claude Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp kriszyp marked this pull request as ready for review June 24, 2026 14:18
Kris Zyp and others added 3 commits June 25, 2026 18:34
Three stale-auth-cache races caused ~50% CI failure rate with varying line numbers.
alter_role returns HTTP 200 after the DB write but before the async ITC
permission-cache invalidation reaches all worker threads, so the next request
can hit a worker with stale permissions. Affected spots:

- CSV upsert with newly granted attribute permissions
- Two cross-schema SQL JOINs that require the updated role to be visible

Each affected request is now wrapped in the existing waitFor() poll helper,
retrying until the permission change is visibly effective.

No product code changed. Test-only stabilization. Verified locally 5/5 runs (560 pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fails clearly

Addresses Gemini review: waitFor returns the last value on timeout; assert the
expected status (200/403) before body assertions so a propagation timeout surfaces
a clear failure instead of a confusing TypeError or silent pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous poll predicate (status === 403) was satisfied at an
intermediate/stale role-propagation state — a 403 with empty
unauthorized_access and only one invalid_schema_item — so the test still
flaked after the waitFor was added (CI on the fix commit still failed:
"0 !== 1" on unauthorized_access.length). alter_role propagates to worker
auth-caches via async ITC, and a 403 alone does not prove the *current*
restriction is fully visible.

Poll until the full expected shape is present (1 unauthorized table = dog
AND all 3 invalid_schema_items) before asserting. Verified locally 3/3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kriszyp kriszyp force-pushed the kris/flake-northwind branch from 86aa495 to c0b256b Compare June 26, 2026 00:39
@kriszyp

kriszyp commented Jun 26, 2026

Copy link
Copy Markdown
Member Author

Rebased onto current main (v5.1.14) and fixed an incomplete poll predicate that was leaving this PR's own CI red.

The ALL RESTRICTED complex CROSS 3 SCHEMA JOINS test was still flaking after the waitFor was added — the post-fix run on 86aa495a5 failed with 0 !== 1 on unauthorized_access.length. Root cause: the poll used until: status === 403, but a 403 also appears at an intermediate/stale role-propagation state (empty unauthorized_access, only one invalid_schema_item). A 403 alone doesn't prove the current restriction is fully visible across worker auth-caches.

Now polls until the full expected shape is present (1 unauthorized table = dog AND all 3 invalid_schema_items) before asserting. Verified locally 3/3 (560 pass, 0 fail) on the northwind suite.

Note on remaining red shards: those failures are in other test files sharing the shard (e.g. blob describe_all schema-propagation race, ops-port-isolation WebSocket timeout), not in northwind.test.mjs. They're separate flakes — the schema-propagation one shares this PR's async-ITC root cause, which I've written up as a follow-up issue for a durable fix.

— Claude Opus 4.8

@kriszyp

kriszyp commented Jun 26, 2026

Copy link
Copy Markdown
Member Author

Follow-up issue for the durable product-side fix: #1497 (cross-worker ITC propagation barrier). — Claude Opus 4.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant